{
    "cells": [
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "3b05af3b",
            "metadata": {},
            "source": [
                "(tune-comet-ref)=\n",
                "\n",
                "# Using Comet with Tune\n",
                "\n",
                "<a id=\"try-anyscale-quickstart-tune-comet\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-comet\">\n",
                "    <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
                "</a>\n",
                "<br></br>\n",
                "\n",
                "[Comet](https://www.comet.ml/site/) is a tool to manage and optimize the\n",
                "entire ML lifecycle, from experiment tracking, model optimization and dataset\n",
                "versioning to model production monitoring.\n",
                "\n",
                "```{image} /images/comet_logo_full.png\n",
                ":align: center\n",
                ":alt: Comet\n",
                ":height: 120px\n",
                ":target: https://www.comet.ml/site/\n",
                "```\n",
                "\n",
                "```{contents}\n",
                ":backlinks: none\n",
                ":local: true\n",
                "```\n",
                "\n",
                "## Example\n",
                "\n",
                "To illustrate logging your trial results to Comet, we'll define a simple training function\n",
                "that simulates a `loss` metric:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "19e3c389",
            "metadata": {},
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "from ray import tune\n",
                "\n",
                "\n",
                "def train_function(config):\n",
                "    for i in range(30):\n",
                "        loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
                "        tune.report({\"loss\": loss})"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "6fb69a24",
            "metadata": {},
            "source": [
                "Now, given that you provide your Comet API key and your project name like so:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "993d5be6",
            "metadata": {},
            "outputs": [],
            "source": [
                "api_key = \"YOUR_COMET_API_KEY\"\n",
                "project_name = \"YOUR_COMET_PROJECT_NAME\""
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "e9ce0d76",
            "metadata": {
                "tags": [
                    "remove-cell"
                ]
            },
            "outputs": [],
            "source": [
                "# This cell is hidden from the rendered notebook. It makes the \n",
                "from unittest.mock import MagicMock\n",
                "from ray.air.integrations.comet import CometLoggerCallback\n",
                "\n",
                "CometLoggerCallback._logger_process_cls = MagicMock\n",
                "api_key = \"abc\"\n",
                "project_name = \"test\""
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "d792a1b0",
            "metadata": {},
            "source": [
                "You can add a Comet logger by specifying the `callbacks` argument in your `RunConfig()` accordingly:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "id": "dbb761e7",
            "metadata": {},
            "outputs": [],
            "source": [
                "from ray.air.integrations.comet import CometLoggerCallback\n",
                "\n",
                "tuner = tune.Tuner(\n",
                "    train_function,\n",
                "    tune_config=tune.TuneConfig(\n",
                "        metric=\"loss\",\n",
                "        mode=\"min\",\n",
                "    ),\n",
                "    run_config=tune.RunConfig(\n",
                "        callbacks=[\n",
                "            CometLoggerCallback(\n",
                "                api_key=api_key, project_name=project_name, tags=[\"comet_example\"]\n",
                "            )\n",
                "        ],\n",
                "    ),\n",
                "    param_space={\"mean\": tune.grid_search([1, 2, 3]), \"sd\": tune.uniform(0.2, 0.8)},\n",
                ")\n",
                "results = tuner.fit()\n",
                "\n",
                "print(results.get_best_result().config)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "d7e46189",
            "metadata": {},
            "source": [
                "## Tune Comet Logger\n",
                "\n",
                "Ray Tune offers an integration with Comet through the `CometLoggerCallback`,\n",
                "which automatically logs metrics and parameters reported to Tune to the Comet UI.\n",
                "\n",
                "Click on the following dropdown to see this callback API in detail:\n",
                "\n",
                "```{eval-rst}\n",
                ".. autoclass:: ray.air.integrations.comet.CometLoggerCallback\n",
                "   :noindex:\n",
                "```"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.7"
        },
        "orphan": true
    },
    "nbformat": 4,
    "nbformat_minor": 5
}
